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ABSTRACT - , - 

Learner Verification ^nd Revision \LYB) Process of 
Instructional Materials is an ongoing effort for the ^improvement of 
instructiqnal materials based on systejmatic feedback from learners 
who have used the materials. This evaluation "gives publishers a 
method of identifying instructional st3^engths and Weaknesses of a 
product and provides an opportunity to revise that product. LVR 
identifies trouble spots or the absence of them, since the only 




es 

are carried out according to a flexible format. The format includes 
^a) descriptive information on the pro^uct^ (b) instructional design, 
(c) intended learner outcomes to be jlnve'stigated, (d) conditions of 
use of product, (e) techniques 'for gathering feedback, (f) 
description of learners used in LVR brpcess, (g) analysis of 
findings, (h) specific improvements niade^ (i) background and future 
of product. Three appendixes and a review of Task Force reactioiis are 
included. (DS) 
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This document - resulting from a National LVR Task Force 
sponsored by EPIE Institute - is distributed in the public interest by 
EPIE Institute, a not-for-profit, cost-sharing, consumers' network 
serving selectors and users of instructional materials and equipment. 
The LVR Guidelines are offered to educators and to publishers and 
other producers of instructional materials for use or adaptation in 
light of their needs. It is requested that those using or adapting the 
LVR Guidelines send feedback for the revision of the Guidelines to 
the National LVR Task Force, in care of: 



EPIE Institute, 463 West Street, New York, New York 10014 [212-675-1163] 



or 



CONTENTS 



Preface - 

I n trod ucti on. 

Orientation Tovyard a Definition of the LVR Process 

Guidelines for Reporting and Assessing LVR Activities 

"Appendix i; , ^ 

Task Force Reactions ......1 



\ 



. PREFACE 



» As' a journalist put it recently, "Not being in favor of using the process of 
learner verification and revision to improve instructional piateriais is like not 
being in favor of God and motherhood; the problem is how it ought to be 
done." 

The following pages contain the efforts of a Task Force of 36 persons who have 
invested a considerable amount of tlieir time and expertise since last December v 
to the task of helping those who are serious about utilizing the process of LVR 
to do so during the year ahead. The Pilot LVR Guidelines that resulted from 
the work of this TaSk Force are, like instructional matejials themselves, subject 
to improvement. And also like instructional materials themselves, they are not 
"the only way" .to improve student learning. Indeed, these Guidelines will be 
improved through feedback fr<3m "pilot" user^. In fact, they are scheduled for 
revision by members of the Task Force after tlie Guidelines - or adap.tations of 
them - have been applied by educators and educational materials companies 
for a year or so. ' 

Because virtually all those who contributed to the work of the Task Force have 
agreed to serve once again - in reviewing and revising in order to improve these 
Guidelines - the usual expression of "thanks 'for a job well done" is, in this 
instance, more appropriately phrased as "thanks for a job well begun." 
Therefore, we at EPIE Institute express this more appropriate sentiment to the 
students (particularly the students, since we all .work for them); teachers; 
administrators; local, state, and national officials; school boatd members; 
educational researchers; and representatives of the educational materials 
industry who have contributed thus far to the work of the National LVR Task 
Force. We thank you for your continued commitment and we look forward to 
the continuation of this important work. 



P. Kenneth Komoski 

Executive Director, EPIE Institute 

6 June 1975 



4 



INTRODUCTION 



For some years now, instructional materials developers have advocated and employed 
"formative evaluation" procedures as a means of improving the instructional effectiveness of 
products prior to their entering the commercial market. These procedures can now be 
recognized as irriportant early steps toward the broader concept of learner verification and 
revision of instructional materials. The word-cluster, "learner verification and revision," was 
first used to describe the process of continual product improvement on the basis of feed- 
back from learners in testimony before a subcommittee of the House of Representatives in 
1971: " In addition to these new departures, the [instructional materials] developers also^ 
took a new approach to the development of their materials: they thoroughly evaluated the " 
effectiveness of their materials on a group of learners and revised them accordingly, before 
they were released for commercial distribution. This process of learner verification and 
revision ..." [More extensive excerpts from this Congressional testimony appear in Section 
A of the Appeiidix.] 

The following year, the legislature of a large state enacted a law requiring publishers 
submitting instructional materials for adoption to "develop plans to improve the quahty and 
reliabihty of instructional materials through learner verification." Later,, another influential 
state made it a legal requirement for publishers to "provide written proof of the use of the 
learner verification and revision process." [The complete jelevant portions of the statutes 
appear in Sectioi?^ B of the Appendix.] A leading city'then announced that it would pilot 
LVR requirements for a year, a leading state educatix)n department announced that it 
wanted "written evidence [of] . . . how the publisher (authors) used tlie results of the field 
trial or pilot test in completing the book," and it became known that other states and local 
education agencies were interested iii, even if confused by, the LVR Process. 

Such was the momentum that by the fall of 1974, Education US. A. warned that the LVR 
Process "could cause chaos." Reporting fears expressed by members of an education 
industry associafion, the newsletter stated that "only a handful of people have any idea ^ 
what learner verification is and the rest are hopelessly confused and misinformed. ... If 
some understanding and agreement is [sic] not brought to this subject, tlie industry could 
face 50 different guidelines — one for each' state." 

In an attempt to bring "understanding and agreement ... to this subject," EPIE Institute 
organized and sponsored a National LVR Task Force charged with developing guidelines for 
publishers and educators. The Task Force is composed of 36 "official" and "unofficial" 
parficipants and 12 Observers. The composition of the Task Force represents just about 
every relevant educational, commercial, and lay interest: program developers from the 
educational materials industry together with educational researchers from unn ersity centers; 
students, teachers, and people from citizens' and other interested groups, sales people from 
the educational materials industry together with people from tlie state and local level who 
are responsible for selecting instructional materials. 

The paper that follows - with its wordy titles, "Orientation Toward a Definition of the 
LVR Process" and "Guidelines for Reporting and Assessing LVR Activities" - is tlie result 
of four Task Force meetings over a three-month period. Precisely put, the paper that follows 
is the result of EPIE Institute's perceptions of the Task Force's collective position. 



It must be understood that the guidelines are not intended to be- viewed as model legislatipn. 
They are not even intended to be viewed as model regulations or policies. In fact, they are 
not even — nor may they ever be — in final form. They are best viewed, as one Task Force 
participant has said, as "the beginning of a series of successive approximations toward an 
ideal — as yet undefined — that points toward the continual improvement of the tools of 
learning." 

In short, the following "Orientation Toward a Definition of the LVR Process" and 
"Guidelines for Reporting and Assessing LVR Activities" are offered in the pubHc interest, 
as guidance for any interested party to accept, to adapt, or to reject. The Task Force — 
which, it is expected, will reconvene after a year or so of experience testing the guidelines — 
has no authority save the reasonableness, credibility, and usefulness of its efforts. • 



Kenneth E. Baranski 
Consultant to EPIE Institute and 
Secretariat to the Task Force 
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ORIENTATiON TOWARD A DEFINITION OF THE 
LVR PROCESS 



The LVR Process is a learner-based process for improviiig instructional materials. 

Stated in a more definitional manner: The LVR Process of Instructional Materials is an 
ongoing effort at the improvement of instructional materials based on systematic feedback 
from — as the primary source — learners who have used the materials. 

In other words, with the LVR Process, publishers decide whether to revise their materials - 
and specifically how to revise them, if they are to be revised — on the basis of data from 
learners. These data can be collected from various people: directly from the learners 
themselves; from teachers who report their perceptions of learners in their classrooms; from 
staff members of publishing companies; and from other observers of learners using the 
materials. Also, these data can be collected in various ways: through classroom observations; 
through interviews with learners, individually and collectively; through interview's with 
teachers; through structured questionnaires filled in by learners and teachers; through 
criterion-referenced tests; and so on. And, finally, these data are assessed by publishers and, 
perhaps, by teachers and others. 

It is necessary to stress, because there is evidence of misunderstanding on this point, that the 
LVR Process does not guarantee or certify that instructional materials will bring about 
explicitly predictable results with learners. In fact, responsible publishers, notwithstanding 
any field-testing they may have done, do not make such guarantees about their materials, 
nor, indeed, do responsible buyers expect such guarantees. It is obvious that there are so 
many variables operating in the learning process that such guarantees cannot honestly be 
made and, for this reason, ought not be made. [At this point, some amplification, which can 
be found in Section C of the Appendix, may be helpful.] 

Lest there be another sort of misunderstanding about the specific role of the LVR Process in 
relation to the overall publishing process, it should be understood that publishers, in 
developing and revising their instructional materials, engage in a number of activities aimed 
at insuring product quality. For example, accuracy and currency of subject matter are a 
primary consideration during^ the development and revisi6n of certain instructional 

^ niaterials; publishers can turn to content specialists in order to insure accuracy and currency 
of such subject matter. Similarly, when it comes to fair 'and full treatment of important 
social issues, publishers can turn to appropriate experts and groups for review and reaction. 
But when the issues are how well learners understand the subject matter and other learnings 

. dealt with in specific instructional materials, and what particular aspects of those materials 
might be revised in order to improve the understanding of learners, then publishers must 
turn to data gathered from learners. Having gathered such data, publishers will be able to 
identify instructional strengths and weaknesses of a product and to revise, if necessary, 
specific aspects of a product in order to improve its instructional effectiveness. This 
learner-based approach to instructional improvement is tlie specific role assigned to the LVR 
Process. 

It may seem reasonable to assert that all materials marketed for use by learners should be 
subject to the LVR Process. But because not all materials used by learners have been 
explicitly designed for the purpose of instruction, there is a need to clarify which materials 
are candidates for the LVR Process and which are not. First of all, what are instructional 
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: materials? One incomplete definition can be tentatively advanced: Instructional materials 
are those materials which have been explicitly designed to enable learners to achieve 

^ specifically stated, or otherwise clearly apparent, instructional intents. An algebra textbook, 
a filmstrip.on the metric system, and television lessons on art history are all obviously 
instructional materials under this definition and, thus, clearly candidates for LVR. A work 
like Macbeth would not be a likely candidate for the LVR Process, because it contains no 
explicitly apparent instructional intent. Although Macbeth is used in classroorris to achieve 
certain instructional intents, those intents are brought to tlie play by the teacher. Yet, a 
publisher conceivably might want to investigate the suitaoility of the study of Macbeth in a 
certain grade or to gather data, for example, on affective effects of the play on learners; 
while revision of Shakespeare's text obviously would not follow, what might be revised in 
the light of such feedback are the supporting instructional materials: reading guides and 
manuals used by learners and teachers in studying the play. Also, a modernized or otherwise 
rewritten version of Macbeth, which would constitute a Macbeth that is no longer 
Shakespeare's, could become a candidate for the LVR Process. It seems safe to aver that if a 
publisher describes materials as instructional when they're offered for sale — or if selectors 
view them as instructional materials - those materials should be considered candidates for 
the LVR Process. 

Therefore, from the perspective of the LVR Process, it is helpful to view the universe of 
instructional materials as a continuum. All materials used m schools can be placed at some 
point on this continuum, with respe^ct to the degree of their instructional design and tlieir 
instructional intents. Some programs, like basal reading programs, have very obvious designs 
and very clear intents. Others, like the actual text of Macbeth standing by itself or a series of 
slides of great paintings by themselves, may have no obvious instructional design or no 
clearly apparent instructional^ intents. Before submitting a LVR Report for a product, a 
publisher should anticipate the expectation of "substantial LVR," as it were, for products 
with maximum iiistructional design and intents. For products of Uttle or no built-in 
instructional design or apparent instructional intents, a publisher may wish to substitute for 
substantial LVR a documented statement supporting the notion that the LVR Process does 
not reasonably apply to those products. Such a supporting statement will perform two 
important functions: (1) It will enable the publisher to make undeniably clear the nature 
and intended role of the product, and (2) it will enable the purchaser to assess better tlie 
potential uses and instructional expectations of the product. 

In summation: As educational materials increase in structure and in the explicitrtess of their 
instructional intents, LVR activities increase in importance and in appropriateness; as 
educational materials have less structure and are less clear in their intents, LVR activities are 
less important and may even be replaced by a well argued statement that the LVR Process is 
either impossible or inappropriate with a specific product. 

The issues of the selection of learners for the LVR Process and who conducts LVR activities 
are too important to be ignored at this point, particularly because they can be briefly dealt 
with. As President Lincoln reporte^lly instructed his tailor to make his trousers long enough 
to reach the ground, so publishers should select enougli learners for the LVR Process to do 
the job. A less flippant explanation is given in the following "Guidelines for Reporting and 
Assessing LVR Activities," in the section treating the description of learners used in the 
LVR Process. Witli respect to who conducts LVR activities, the issue should not be an issue 
because it should be evident that only the publisher should conduct LVR activities. The 
notion of an outside agency's conducting LVR for a publisher reveals specious reasoning. 
LVR is not a certification process which aims for some sort of seal of approval from an 
independent testing agency; LVR is a process for identifyiiig trouble spots (or the absence 
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•of them) and the only appropriate people to engage in it are the people intimately involved 
in developing the product. 

It Should not be iniferred from anything said above that LVR is in the exclusive domain of 
publishers. While only publishers can initiate the decision to carry out to completion the 
LVR Process, the responsibility for LVR Qertainly does not apply Only to publishers. 
Schools share this responsibility. Whereas publishers accept the responsibility for product 
development and revision and for the specific techniques to be used in gathering learner 
feedback, schools accept the responsibility for arranging for the participation of learners and 
teachers in the LVR Process. If a school expects that a publisher has put materials through 
the LVR Process, that very same school must be ready to make it possible for the publisher 
"to LVR" the materials, which includes the working out of logistics and necessary safe- 
guards for LVR activities. Furthermore, schools themselves can take tlie initiative (as some 
are now doing) in devising data-gathering techniques, in using the techniques to gather data 
from learners, and in reporting the data, perhaps with their analyses, to publishers. 

In otlier words, any educational unit - whether an individual classroom, a single school, a 
school district, or a state - that makes LVR a requirement of publishers must itself make a 
commitment to LVR. And this commitment, in addition to agreeing to cooperate with 
publishers, includes taking or delegating the responsibility for assessing LVR Reports from 
publishers. 

The "Guidelines for Reporting and Assessing LVR Activities," which follow, should not be 
viewed by purchasers as a panacea for the continued improvement of the learning effective- 
ness of instructional materials. Schools must keep in mind that effective instructional 
materials require effective use in the classroom. The LVR Process is but one activity, albeit 
an important one, directed toward the continued improvement of instruction through the 
development, selection, and use of increasingly effective instructional materials for learners. 
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GUIDELINES FOR REPORTING AND ASSESSING 
LVR ACTIVITIES 



The purpose of a report on LVR activities is to enable the publisher- of instructional 
materials, to provide educators with evidence that a particular product has been - or soon 
will be - put through the LVR Process as part, of a continuing effort to improve its 
effectiveness with learners. Although a publisher can fashion various LVR models and yet 
conform to the basic objective of the LVR Process, certain elements in every model are 
essential. Tliese essential elements, which taken together operationally define the process, 
are asterisked below and should be addressed in every LVR Report. What is offered below, 
then, is not an inviolable /orw that all publishers must use ot that purchasers demand be 
used; rather; it should be viewed as a flexible /orwat which constitutes a set of gtridelines to 
the pubUsher for preparing a LVR Report and to educators for reading and assessing such a 
report. 

Descriptive Information on Product 

[The publisher should provide "catalog"' information on the product, such as title, 
authors, copyright, medium or media invcilved, kinds of supplementary components, and 
so on. The "product" is any kind of ma :erial offered for adoption consideration as an 
instructional material. It should be desciibed as it nomially is when submitted to an 



adoption/selection committee, that is, a 



six-year textbook series, for example, or a 



UVAW^ l-AVrJfc/ UVAVW •-.WAl — -7 7 _ _ , _ 

multicomponent (and perhaps multimedia) "system" being considered as one product.] 



Instructional Design 

[The "instructional design" of a curriculum product can be thought of as the overall 
learning plan for the product. The desiga should be described and the underlying 
rationale for that design should be stated. 

[It is to be expected that a product with integrity of design - a product whose instruc-. 
tional elements are well integrated - will easily be put through a well designed LVR 
effort. Conversely, a product without an apparent "instructional design cannot easily, if at 
all, be subjected to a cycle of the LVR Process.] 

Intended Learner Outcomes To Be Investigated 

[This should be 'a statement of what the publisher intends the product to accomplish 
when used with integrity with learners. If such a statement appears in the product itself 
or in accompanying proniotional pieces, a reference may be made to where it can be 
found. 

[It may be that at a particular time the publisher wants to investigate only certain 
outcomes within the larger scope of intended learner outcomes, in which case this 
intention should be stated. It may be also that a publisher of a six-year textbook series 
(with various attendant materials), for example, may investigate selected outcomes of an 
exemplary portion of the entire program and use the resultant data to guide revisions of 
other portions of the program; if so, this should be made clear.] 
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* Conditions of Use of Product 



[The publisher should specify the overall instructional setting intended for the product. 
Basicdly, this includes a description of what is termed tlie **target population" and a 
noting cf the kinds and extent of ^'t'^acher preparation." The target population is simply 
the kinds of learners the prodtict is intended for. Characteristics which may or may not 
be relevant are:' grades students are assigned to because of their maturity, not achieve- 
ment, levels; actual grade levels, particularFy with respect to reading achievement; ethnic 
backgrounds; family and community socioeconomic level; :md so on. Teaclier preparation 
involves such teacher-related matters as supplementary teaclier materials, inservice teacher 
education, and classroom preparation.] 



* Techniques for Gathering Feedback 

[Techniques for gathering feedback from learners may include, but are not limited to, 
individual learner and classroom observations, interviews, questionnaires, and 
criterion-referenced tests. These tecliniques should be designed to gatlier data both on 
direct-learning effects and on affective reactions c|f learners. 

[Techniques used in large-scale, field-test validation attempts are not to be confused with 
techniques used for gathering learner feedback fot tlie LVR Process. Similarly, techniques 
used to elicit product testirrionials for promotional purposes Hke this textbook very 
much and so does my class" ) ~ as opposed to reactions gatliered as a result of question- 
naires or interviews, whicli may be used in promotion pieces - are inappropriate for the 
LVR Process.] 



* Description of Learners Used in LVR Process 

[The selection of learners for tlie LVR Process is not to be confused witli a scientifically 
drawn "national sample'' of students required for the validation of standardized tests and 
for attempts at validation of other materials. Such a ^'sample'' will perforce involve 
thousands of students. A LVR selection because^ the aim of LVR is to identify trouble 
spots in order to improve materials, not to prove how well materials work - may 
comprise a handful of learners from the target population or a Iiandful of classes. In fact, 
in some instances - and the publisher should justify the use of this procedure - a few 
students might be taken tlirough tlie LVR Process in the offices of tlie publisher, unless 
there are certain conditions of a classroom setting which are nec'essary and which cannot 
be approximated in the offices. 

[The factor of conditions of a classroom setting suggests tlie relevant factor of two 
interrelated bases of product performance subject to LVR improvements, whicli can be 
conveniently distinguished as "textual'' and '^contextual" characteristics. Textual 
characteristics are characteristics of tlie materials themselves which learners use (goals and 
objectives: scope and sequence: provisions for evaluation; and so on); contextual 
characteristics are characteristics of the setting the materials are used in (the presence or 
absence of teaclier's editions or manuals, or certain other supplementary resources; tlie 
teaching/management plan; and so on). Students miglit be taken to tlie publislier's office 
if textual improvements are being souglit by the publisher, but tliey probably should not 
be if contextual improvements are being souglit, 

[In summation, the publisher should report: (1) the procedure for .selecting learners, (2) 
the relevant characteristics of tliese learners, and (3) the rationale for tliis step. 
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[Those charged with the responsibility of assessing LVR Reports are urged not to get 
involved in a "numbers game" - that is, ranking publishers according to the number of 
learners used in a publisher's LVR activity - because such' a ranking is at best un- 
productive, at worst grossly misleading.] 



* Analysis of Findings • 

[After assessing the extent to which a product is and is not as effective as it might be with 
a selection of learners and after gathering data, the publislier analyzes the data. It is 
desirable that the publisher make known the analysis of the findings in a LVR Report. It 
may be, however, that the publisher would consider such information to be proprietary, 
because reyealing it could give competing publishers an advantage; in such cases, the 
publisher ipay explain the reasons for keeping the findings and the analysis of the findings 
confidential, and, with this condition accepted by afi adoption committee, release the 
infonpation only on request. It may be also that the publisher states in a LVR Report a 
justification for not releasing any information whatsoever on findings, in which case the 
publisher sliould also describe - with as much specificity as prudence allows - the kinds 
of trouble spots identified both within the materials used by learners and in the overall 
teachiri^managenient plan, 

[Here too, those charged with tlie responsibility for assessing LVR Reports should 
understed the competitive nature of educational publishing, and they should be 
influenced by the reasonableness of the report that the publisher does provide,] 



* Specific Improvements Made 

[The analysis of findings enables the publisher to identify exactly where specific changes 
should be made, both textual and contextual changes. Improvements can involve aspects 
relating to learner materials (textual), such as verbal or visual communications, manage- 
ability, appeal to learners, goals and objectives, scope and sequence, various kinds of bias, 
motivational elements, directions to learners, activities for learners, ^nd the congruence of 
product elements. Improvements can relate to teaching aspects (contextual), such as 
classroom preparation, record-keeping provisions, teaching design, and inservice edu- 
cation. ■ 

[As an ideal component of a LVR Report, the very in-house record of changes compiled 
by the publisher - including, perhaps, an annotated version of the product used with the 
selection of learners - might be made available by the publisher, on special request, for 
examination by an adoption committee, 

[Once again, however, the proprietary nature of changes made may constrain a publisher, 
in which case the reasonableness of what the publisher does repOrt is itself a factor in 
assessment. Here, though, even more is involved than what was noted above, A publisher 
may fear that citing a large number of changes might be an adverse reflection on the 
product and the publisher. Then, too, an unscrupulous publisher might be tempted to 
develop an intentionally shoddy product to demonstrate, through the LVR Process, a 
substantial quantity of improvements. 

[It is, therefore, up to those assessing LVR Reports to be conscious of realities constrain- 
ing - and of apprehensions cautioning - a responsible publisher, and to be mindful of 
trickery tempting an irresponsible publisher.] 
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Background and Future of Product 

[The publisher should briefly ^describe any part of tlie product's history that might be 
relevant and helpful to a full understanding of ^e publisher's LVR efforts with the 
product. Abo, the publisher may waiit to describe plans for future LVR efforts with the 
product, although the publisher would want to consider that a premature revelation of 
plans might giye competitors an advantage.] 



/ Person to Contact for Clarification, More Detail, or Updated Information: 

I ■ ■ ■ ■ 
Namb Position 

: 1 Division 



Company 
Address 



Telephone 
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APPENDIX: Section A | 



Hearings before the Select Subcommittee on Education of the Committee on.Education and 
Labor; House of Representatives; Ninety-Second Congress; First Session on H,R. 33, H.R. 
3606, and Other Related Bills To Establish a National Institute of Education, and for Other 
Purposes; 1971: 

"The materials I [P. Kenneth Kompski,. Executive Director, EPIE Institute] refer to are 
the textbooks, films, tapes, kits, and complex multimedia systems that are the concrete 
artifacts of learning in our schools. These materials are the tools of learning, constantly 
used and depended upon by teachers and students alike. Indeed, one can conceive of a 
school without walls, or even an educational system without schools, but a modem 
educatjdnal system without tools - in the form of a variety of educational materials - is 
inco;i^eivable. Today's education requires that these tools be better, sharper, more 
de|/endable — in other words, more effective — than ever before. ... 

.'"In addition to these new departures/ the developers also took a new approach to the 
development of their materials: they thoroughly evaluated the effectiveness of their 
materials on a group of learners and revised tliem accordingly, before they were released 
for commercial distribution. This process of learner verification and revision emerged 
about a decade ago, primarily in^ the development of programmed instruction materials. 
Since that time the process has been adopted by many FederaUy:funded product develop- 
ment programs, but by very few comme^rcial producers. 

"One reason why Commercial producers have not adopted the verification process may be 
that few schools, look for evidence of leamer-verification when selecting curriculum ^ 
materials. . . . Another reason is that new materials with new approaches are harder to 
seU. 

"... These ^terms (learner verification , and revision] are simply a researchers way of 
saying that the learning effectiveness of a product ^ill be improved if it is taken through a 
systematic cycle, of tryouts with learners followe,d by revisions based on the feedback. 
Such evaluations need not always involve large groups of learners. Through appropriate 
sampling ^ small group of 'target' students can give the product developers ample 
opportunity to catch errors and trpuble spots and to revise accordingly 

"[In} evaluating products, school selection committees must devote most of their limited 
time to judging a product's content and pedagogical approach. Practically no time can be 
given to gathering evidence of a material's learmng;effectiveness. Committees assume that 
' materials with 'good content and the right approach' will, by definition, be effective with 
le^ners. Logical as this may sound, litis not necessarily tme. . . . ^ 

"Fortunately, some research exisl^ which has examined the question of whether it is 
possible to infer the learning effectiveness of particular examples of instructional 
materials by simply examining therd. This research raises serious doubts about the reliabil- 
^^ -^ ity of the practice of judging the quality of learning effectiveness without the help of 
leamer-verification data. ... • 
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"Not just existing research, but common sense, makes it quite clear that at present any 
responsible effort to create- or select materials of proven learning effectiveness must use 
data from lcamer-verific|ition. So the problem is not that research does not answer the 
question directly and unj^biguously - the real problem is that the question of learning 
effectiveness does not sekm to be of great interest to the producers and purchasers of 
educational materials." 1 
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APPENDIX: Section B 



California Instructional Materials Law; Compiled by Curriculum Frameworks and In- 
structional Materials Selection Unit; California State Department of Education; Sacramento, 
1974: 

"9234. 'Learner verification' means the continuous and thorough evaluation of in- 
structional materials for^heir effectiveness with pupils. 

"9426. P»iblishers and manufacturers shall, in accordance with rules and regulations 
adopted by the state board, develop plans to improve the quality and reUability of 
instructional materials through learner verification. Governing boards shall be encouraged 
to permit publishers and manufacturers to have limited access to classrooms for necessary 
testing and observation. Publishers and manufacturers shall provide copies of test results 
and evaluations made as p^ of learner verification at the request of any governing 
board." 



Unofficial; Chapter 233; Courses of Study and Instructional Aids; Compiled by TextbooK^ 
Services Section, Division of Elementary and Secondary Education, Florida Department of 
Education; 07/09/74: 

"233.25 . . .Publishers and manufacturers of instructional materials or their represen- 
tatives shall: 

. . . (3) Submit, at a time designated in section 233.14, Florida Statutes, the following 
information: . ^ 

"... (b) Written proof of the use of the learner verification and revision process during 
prepublication development and post-publication revision of the materials in question. 
For purposes of this section learner verification is defmed as the empirical process of data 
gathering and analysis by which a publisher of a curriculum material has improved 'the 
instructional effectiveness of that product before it reaches tlie market and then 
continues to gather data from learners in order to improve the quality and reliability of 
that material during its ftill market life. Failing such proof, if the publisher wishes to 
submit material for adoption, he must satisfy the state instructional materials selection 
council that he will syst^maticall^y gather and utilize learner-verification data to reviie tlie 
materials in question to better m>et the needs of learners throughout the state. Such text 
revision should be interpreted as^including specific revision of the materials thenjselves, 
revision of the teachers' materials and revision of the teachers' skill through-retraihing, it 
being the intent of the legislature that ieamer-verification and revision data shaUAncIude 
data gathered directly from learners and that such data may include the rbfeults of 
criterion-referenced and group-normed tests, direct learner comments, or information 
gathered from written questionnaires from individual or small group interviews^ and not 
precluding the use of secondary data gathered from teachers, supervisors, parents, and all 
appropriate participants and observers of the teaching-learning process." J 
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Unofficial; Proclamation of the State Board of Education Advertising for Bids on 
Textbooks; Texas; May 1975: 



"1.13 In the developmental stage of each textbook, authors should have field tested the . 
material with teachers and students in planned field trial or pilot test. Written evidence 
from such testing, showing how the publisher (authors) used the results of the field trial 
or pilot test in completing the book, should be made available to the State Textbook 
Committee.** 



/ 
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APPENDIX: Section C 



There appear to be at least three arguments which mUitate against the use of the term, 
"learner verification and revision," making its handy abbreviation, "LVR," desirable. 

First, there seems to be a tendency for some people to refer to "learner verificati'm," rather 
than "learner verification and revision." Althougli the shortened version may be more 
comfortable to use, its usage can so mislead people that the crucial point of the LVR 
Process - that it is a process for revising materials - can be irrevocably lost. 

Second, the word, "verification," apparently suggests to some people tlie concept of 
"proof." Thus, in the context of instructional materials, these people expect from the LVR 
Prbcess some sort of certified proof tliat instructional materials will achieve certain 
predictable results. Here too, it is crucial to understand that the LVR Process is not 
intended to and does not — produce such proof. / 

Third, there is evidence • that some people feel "verification" and "validation" are 
synonymous. Traditionally in the field of education, "validation" has been used to mean the - 
deteWining of achievement norms, particularly with standardized tests. Once again, it is 
cruciai to understand that the LVR Process makes no such determination. 

If these arguments dissuade people from using the term, "learner verification and revision,"/ 
perhaps arguments should be advanced against the abbreviation, "LVR." A number of 
people have suggested that since it has been stated at the outset that "The LVR Process is a 
learner-based process for improving instructional materials," the obvious label is "L-BR," 
standing for "learner-based revision." When it reconvenes after a year's experience, the Task 
Force may be discussing "L-BR Guidehnes." 
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TASK FORCE REACTIONS 



Those attending Task Force meetings were sent *'LVR Paper: Draft #5" on 21 March 1975 
and were invited to select one of three options. The options and the reactions follow. 



option (a) I recommend that Draft #5 of the LVR Guidelines in its present 

form, or with the minor emendations I have made, be distributed to 
interested parties. 

Option (b) I am not completely satisfied with Draft #5 of the LVR Guidelines 

and ask that the statement of my reservations, which Tm attaching, 
be appended to it when it is distributed to all interested parties. \ 

option (c) I am totally dissatisfied with Draft #5 of the^tVR Guidelines and 

wish that my dissatisfaction, as I have expressed it in the attached 
statement, be appended to it when it is distributed to all interested 
parties. 



V 



Tracy Archibald 
[high school student] 

c/o Executive High School Internships Program 
Wichita Public Schools 
640 North Emporia Street 
Wichita, Kansas 67214 

selected option (a) 



Robert Baker 

[educational researcher] 

Director of Planning and Program 

SWRL Educational Research and Development 

4665 Lampson Avenue 

LosAlamitos, California 90720 

selected option (a) 



Donna E. Canick » 
Research Coordinator 
Marketing - Program Planning 
Encyclopaedia Britannica EducationaHJorporation 
425 North Michigan Avenue \ 
Chicago, Illinois 60611 \ 

[For option selected, see 
listing for Suzanne T. Isaacs.] 



Myma Cooper 

Special Representative for Educational Services 
American Federation of Teachers (AFL-CIO) \ 
260 Park Avenue South 
New York, New York 10010 

no response received 
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Sterling P, Davis 

Area Coordinator, Sales Department 

Guidance Associates 

757 Third Avenue 

New York, New York 1 001 7 

[For option selected, see listing for 
Suzanne T. Isaacs; also, see attachment #1. 



V 



Terry Denny 

/ educational researcher] 
Professor of Education 
College of Education 
University of Illinois 
Urbana, Illinois 61801 

selected option (a) 



Emory Dykes 

Administrator, Textbook Services 
Florida Department of Education 
TalMiassee, Florida 32304 

[See attachment #2: "Guidelines for the Learner Verification 
and Revision Process; 1976 Adoption; State of Florida, 
Department of Education."] 



Estelle Fuchs 

Professor, Graduate Center 
Qty University of New York, and 
Professor, Hunter College 
Qty University of New York 
New York, TV^v York 10021 

selected option (a) 



Edythe J. Gaines 
Exeaitive Director 

Division of Educational Planning and Support 
Board of Education of the Qty of New York 
110 Livingston Street 
Brooklyn, New York 11201 

selected option (a) 



Michael Halperin 
Vice President 
Q-ED Productions 
2921 Wost Alameda Avenue 
Burhank, California 91507 

[For option selected, see 
listing for Suzanne T. Isaacs.] 
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Robert J. Harper II 

Director, Model Legislation Project 
Lawyers* Committee for Civil Rights Under Law 
733 Fifteenth Street, N. W, 
Washington, RC 20005 

selected option (a) 



Gary Haseloff 
Chief Consultant 

Division of Instructional Resources 
Texas Education Agency 
201 East Eleventh Street 
Austin, Texas 78701 

selected option (a) [See attachment #3.] 



Carolyn Hawley 

formerly. Coordinator, Ad Hoc Committee to Improve 
North Carolina's Textbook Selection Procedures 
c/o Learning Institute of North Carolina 
1006 Lomond Avenue 
Durham, North Carolina 27601 

selected option (a) 



Wayne Henderson 

Administrative Assistant to the Chief Deputy Superintendent 
California State Department of Education 
721 Capitol Mall 
Sacramento, California 95814 

selected option (a) 



Jesse Holmes 

[high school student] 
c/o Urban Journalism Workshop 
1709 Third Street, NE. 
Washington, D.C 20002 

selected option (a) 



Suzanne T. Isaacs 
Vice President 

Society for Visual Education, Inc. 
1345 Diversey Parkway 
Chicago, Illinois 60614 

option (c) selected by Board of Directors of 

Educational Media Producers Council [See attachment #4.] 



Girvin Eaton Kirk 
Executive Director 

National Council for Children's Media, and 

Research Consultant to Sesame Street Mamzine 

Children's Television Workshop 

One Lincoln Maza 

New York, New York 10023 

selected option (a) 



Susan S. Klein 

Senior Research Associate 

School Practice and Serviqe Division 

Office of Dissemination and Resources 

National Institute of Education 

Department of Health, Education, and Welfare 

Washington, D.C 20208 

selected option (b) [See attachment #5,] 



Marilyn Langdon 

[member. National School Boards Association] 

School Board Member 

Racine Unified School District jfl 

Racine, Wisconsin 53405 

selected option (b) [See attachment #6.] 



James E. LeMay 

Senior Vice President, Marketing 
Coronet Instructional Media 
65 East South Water Street 
Chicago, Illinois 60601 

[For-optiorr selected, see 
listing for Suzanne T. Isaacs.] 

Ruth Mancuso 

President, State Board of Education 
New Jersey State Department of Education 
225 West State Street 
Trenton, New Jersey 08625 

selected option (a) 

Carl L. Marburger 
Senior Associate 

National Committee for Citizens in Education 
Wilde Lake Village Green 
Columbia, Maryland 21044 

selected option (a) 
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Susan M. Markle 
Professor 

University of Illinois at Chicago Grcle 
Office of Instructional Resources 
Chicago, Illinois 60680 

selected option (a) 



Kevin McMahon 
[first-year student] 
University of Vermont 
Burlington, Vermont 05401 

selected option (a) 



Thomas J. Murphy 

Vice President and General Manager 

School Departmen t 

Holt, Rinehart and Winston 

383 Madison Avenue 

Ne\\^ York, New York 10017 

no option selected in statement on behalf of 

Research Committee of Association of American Publishers 

[See attachment #7; also, see attachment #8.] 



Dianne McDonald Newkirk 

/ member, National Education Association] 
Unit Leader and Teacher 
Dogwood Elementary School 
12300 Glade Drive 
Reston, Virginia 22091 

no response received 



Chris Pipho 

Associate Director 

Research and Information Services 

Education Commission of the States 

300 Lincoln Tower 

1860 Lincoln Street . 

Denver, Colorado 80203 

selected option (by [See attachment #9.] 



Donald Senter 

Director, Product Development 

EDLj McGraw-Hill 

1221 Sixth Avenue 

New York, New York 10020 

[For option selected, see 
listing for Suzanne T. Isaacs.] 
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Ned A. Simmons 

Assistant Superintendent 

Administration and Operation 

Monroe County Board of Public Instruction 

Key West, Florida J3040 ■ ' 

[See attachment #2: "Guidelines for the Learner 

Verification and Revision Process; 1976 Adoption; State of Florida, 

Department of Education/'] 



Thomas L. Simpson 

Research and Development Manager 
Chas. A. Bennett Co., Inc, 
809 West Detweiller Drive 
Peoria, Illinois 1614 

coined option (d) ["totally dissatisfied . . . do not 
want the expression of my dissatisfaction appended."] 



Charles Spelker 
Associate Director 

Association for Supervision and Curriculum Development 
1701 K Street, NM 
Washington, DC 20006 

selected option (a) [See attachment #10.] 



Richard C. Spitzer 

Manager, Government Affairs 
Government Information Office 
Science Research Associates, Inc. 
1801 K Street, KW, 
Washington, DC 20006 

selected option (a) 



Deborah Thompson 

[student, Woodrow Wilson High School, Dallas, Texas] 
cjo Executive High School Assistant Program 
Dallas Independent School Districts 
Dallas Chamber of Commerce 
1507 Pacific Street 
Dallas, Texas 75201 

' selected option (a) 



Edwin Thompson 

[member. National School Boards Association/ 
School Board Member 
Tlie School Board of Polk County, Florida 
District if 3 

Biartoyj, Florida 33830 

selected option (c) [See attachment #11.] 
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Robert N. Walker 

Director of Testing 

Ginn and Company 

191 Spring Street 

Lexington, Massachusetts 02173 

no option selected [See listing for Thomas J. Murphy,] 

Edwin Zippe 

[member, National Education Association} 
Teacher, Social Studies Department 
Alexis I Dupont High School 
50 Hillside Road 
Greenville, Delaware 19807 

selected option (a) 



The following people were also present for portions 
of a Task Force meeting. 



Brian Bunch 

Vice President mid Editor-in-Chief 

American Book Company 

450 West 33rd Street 

New York, New York 10001 



Joseph W. Crenshaw / 

Chief, Bureau of Curriculum and Personnel Development 
Florida Department df Education 
Tallahassee, Florida ^2304 



John H. Fisher 
President 

Guidance Associates 

757 Third Avenue 

New York, New York 10017 



Morris Freedman 

Director, Bureau of Audio-Visual Instruction 
Division of Educational Planning and Support 
Board of Education of the City of New York 
131 Livingston Street 
Brooklyn, New York 11201 
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Hannah N. Geffert 
Model Legislation Project 
Lawyers ' Committee for Gvil Righ ts Under Law 
733 Fifteenth Street, W. 
Washington, D.C 20005 



Robert M. McQure 
Program Manager 

Instruction and Professional Development 
National Education Association 
1201 16th Street, N. W. • 
Washington, D.C 20036 



BCMA Associates, unc. 
52 Vmderbilt Avenue 
New York, New York 10017 

Daniel M. Schember 
Model Legislation Project 
Lawyers* Committee for Civil Rights Under Law 
733 Fifteenth Street, N W. 
Washington, D.C 20005 



, Wendell Shackelford 
Chairman 

Guidance Associates 

7^7 Third Avenue 

New York, New York 10017 



Pliilip W, Tiemann 

Head, Course Development 
Office of Instructional Resources 
University of Illinois at Chicago Circle 
Chicago, Illinois 60680 



Kathryn Troy 

Manager, Product Research 

School Department 

Holt, Rinehartand Winston 

383 Madison Avenue 

New York, New York 10017 



Michael W. Walsh 

Coordinator, Sales and Publishing 
American Technical Society 
5608 Stony Island Avenue 
Chicago, Illinois 60637 



Cameron S, Moseky 
President \ 




Memorandum to:' P. Kenneth Kdmoski 

Kenneth E. Baranski 
EPIE 

463 West Street 

New York, New York 10014 

^From: Sterling P. Davis 

Guidance Associates 



21 April 1975 
ATTACagMT #1 

1975 APR 30 (rfll|:35 



. To your memorandum of March 31, 1975 1/ am 

responding individually. Upon careful cohsia^ation , 
I find none of the responses you solicit acceptable. . 
They all lead one to believe. that Draft #5 of the LVR 
guidelines has substance and relevance- in publishing 
and teaching and learning... at least according to the 
varying approbation or disapprobation of all those 
who attended a, National LVR Task Force meeting . You 
are really asking us what we thought' of the dinosaur 
who walked proudy across the plains today, and not if 
we saw the dinosaur • 

■ " . - ■ <> 

The dinosaur's pride I happened to miss, along 
with th^ dinosaur; I /also missed the improvements 
in Draft #5 of the LVR guidelines. I find the whole 
LVR undertaking preposterous. It rears its head . - 

amidst EPIE inferences that educational publishers 
often do a bad job and that they have done a bad job 
for some time. It tampers grotesquely with the con- 
ventional notions of validity and verification in a 
fallacious appeal for workableness and authority. 
It is on the central issue of verification that the LVR 
doctrine falls apart. We must forsake not only 
scientific method but the precious Aristotelian dis- 
tinction between the; particular and the universal. 
This reason alone, I should think, dooms the LVR 
idea to extinction. As an appeal for authority too, 
it looks to educators and state legislatures to mark 
its efforts, while they, quite properly, look back 
with open minds, hoping that LVR will insure better 
quality teaching materials^. It makes so much of 
feedback from the learner ( in Draft #5 practically 
any old learner will do ) that, quite incredulously, 
the learner dwarfs the teacher. 

The wave of consumerism over educational 
publishing,, let loose by EPIE, is as irresponsible 
as it is opportunistic. Among large and small pub- 
lishers, I find the state of the art of making and 
revising educational products far ahead of the for-- 
realistic pretensions of the LVR guidelines. The 
marketplace's judgment is quite final. 



\ 



I 



We have so many institutions in America over 
which the individual has virtually no control that 
it seems to me important to nip in the bud the LVR 
concept, already well nourished in the soil of 
casuistry, lest it become another glob of bureauc- ' 
racy with which to contend. Neither publishers, nor 
teachers, nor students need another rubber stamp. 
And in these depressed times, educational salesmen 
can scarce afford to go dinosaur-watching. 

Si 
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STATE INSTRUCTIONAL MATERIALS COUNCILS ^ 

3* 

Adopted May 2, 1975 



til 

CO 



GUIDELINES FOR ' 
THE LEARNER VERIFICATION AND REVISION PROCESS 
1976 Adoption 
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STATE OF FLORIDA 
DEPAR'mENT OF 'EDUCATION 
TALLAHASSEE/ FLORIDA 

RALPH D. TURLINGTON/ COmiSSIONER 

29 



Position Statement and Guidelines Regarding the Learner-Verification and 



Revision Process of State-Adopted Instructional Materials in Florida 



Florida Statutes, 233.25*- Duties, responsibilities, and requirements 
of publishers and manufacturers of instructional materials Publishers 
and manufacturers of instructional materials, or their representatives, shall: 

(3) Submit, at a time designated in s* 233.14, the following information: 

♦ 

(b) Written proof of the use of the- learner-verification and revision 
process during prepubllcatlon development and postpublication revision of 
the materials in question. For purposes of this section ''learner-verif ication" 
Is defined as the empirical process of data-gathering and analysis by which 
a publisher of curriculum material has improved the instructional effectiveness 
of that product before it reaches the market- and then con1:inues to gather data 
from learners in order to improve the quality and reliability of that material 
during its full market li5e* Failing such proof, if the publisher wishes 
to submit material for adoption, he must satisfy the state instructional 
materials selection council that he will systematically gather and utilize 
learner-verification data to revise the materials in question to better meet 
the needs of learners throughout the state. Such "text revision should be \ 
Interpreted as including specific revision of the materials themselves, revision 
of the teachers^ materials, and revision of the teachers' skill through 
retraining, it being the intent of the legislature that learner-verification 
and revision data shall include data gathered directly from learners, may 
Include the results of criterion-referenced and group-normed tests, direct 
learner comments, or information gathered from written questionnaires from 
individual or small- group interviews, and not preclude the use of secondary 
data gathered from teachers, supervisors, parents, and all appropriate 
participants and observers of the teaching-learning process/ 

POSITION STATEMENT: * * 

The State of Florida Department" of Education and the 12 state Instructional 
Materials Councils participating in the 1976 Adoption recognize that the 
implementation of the Learner-Verification and Re-^Xision Process is not the sole 
responsibility of the Publishers/Producers of Instructional Materials. The 
Schools and the Department of Education of Florida must share this responsibility. 

The Publishers/Producers must initiate the Learner-Verification and Revision 
Process. The Schools and the Department of Education of Florida must arrange 
for the participation of learners and teachers in this process. All concerned 
must cooperate in arranging the necessary logistics and safeguards to develop 
a standardized reporting process. 

The commitment to the Learner-Verification and Revision Process must not be 
limited to the Instructional materials being submitted in the 1976 Adoption, 
but must Include all that may be submitted in the future - Including those 
presently on state adoption. 
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PUBLISHER/PRODUCER GUIDELINES FOR' REPORTS OF COMPLIANCE WITH LVR PROCESS 



There are two approaches. for complying with the learner-verification and 
revision process: 

1. To provide proof of the use of the learner-ver-if ication and revision 
process during prepublication development and postpublication 
revision of the product beirg bid (Proof-Approach Compliance). 

2. To present a plan to systematically gather and utilize learner- 
verification data to revise the product being bid (Plan-Approach . 

• Compliance) . * 

Suggested guidelines for filing written reports for the two approaches follow: 

Proof-Approach Compliance 

1. State in the heading of the report the compliance approach being used. 

2. Give the name of the product being bid. (If the product is a multiple 
item submission such as a series, program, etc., also give the name of 
each component being bid.) 

3. Describe the instructional design of the product and give the rationale 
for the design. . 

4. ' Describe the intended learner outcomes when the product is used properly. 

5. Describe the target population of learners for which the product is intended 

6. Describe teacher preparation needed if the product is to be used properly. 

7. Describe the techniques used for gathering learner-based feedback about 
the product. 

8. Describe the learners participating in the learner-based feedbacjc activities 

9. Give an analysis of the findings from the learner-based feedback that 
indicated product improvement needs. 

10. Give the specific improvements made- that resulted from the learner-baced 
feedback. (The information regarding improvements made^should not be 
limited to learner-used items, but should also include assistance and 
directions for teachers.) 

11. Give, the name, position, company, address, and telephone number of the 
person filing the report. 



Plan-Approach Compliance 



1. State in the heading of the report the compliance approach being used. 

2.. Give the name of the product being bid* (If the product is a multiple 
item submission such as a series, program, etc., also give the name 
of each component being bid.) 

3. Describe the instructional design of the product and give the rationale 
for the design. . 

4. Describe the intended learner outcomes when the product is used properly. 

5. Describe the target population of learners for V7hich the product is intended. 

6. Describe teacher preparations needed if the product is to be used properly. 

7. Describe the techniques that will be used to gather learner-based feedback 
about the product. 

8. Describe the learners who will participate in the learner-based feedback 
activities. 

9. Give the name, position, company, address, and telephone number of the 
person filing the report. 

10. Any publisher using the plan-approach must use the proof-approach when 
resubmitting these or revised instructional materials. 
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Texas Education Agency 




201 East Eleventh Street 
Austin, Texas 



• STATE BOARD OF EDUCATION 
• STATE COMMISSIONER OF EDUCATION 
• STATE DEPARTMENT OF EDUCATION 
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April IT, 1975 



Mr. Ken Saranski 

The Educational Products 

Information Exchange Institute 
463 West Street 
New York, N.Y. 10014 ' 

Dear Ken and Ken, 

This is in response to draft 5 of LVR guidelines. Unfortunately I have 
not had a lot of time lately to attend to this. I did however, ask a few 
other consultants to read these and give me their thoughts. In the midst 
of a letter to you I received your phone call and was pleased to have a 
few more days to read the document again and await any further response 
from Texas Education Agency people. However, I received no other input. 

I did not have many comments to make^this' time, probably because, first, I 
have reacted just to the document and not the philosophies, disagreements, 
and other environmental factors surrounding LVR, and second, because I 
think the document is much better than previously. 

I still think the educational consumer must be responsible for clarifying 
the criteria of LVR and there is still going to be much confusion about 
what LVR means. But this document should help further the concept, put it 
in perspective, and promote a better foundation for understanding. 

I expect some forthcoming firoclamations for Texas textbook adoptions will 
ask for evidence of learner based feedback with the materials. Over a 
period of time we will likely develop more precise criteria for evidence - 
in conjunction with publishers. 

I feel these guidelines are a good start and over a period of years of revi- 
sion we in education, both seller and consumer, will come to understand LVR. 

You must regard my comments and support regarding these guidelines as repre- 
sentative of my opinion, not the Texas Education Agency, even though I 
received no negative comments on the document either from Mr. Lipscomb nor 
Dorothy Davidson. 

• Sincerely yours. 



Gar/ Haseloff, Chief Consultant 
Division of Instructional Resources 
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ATTACIMENT #4 




EDUCATIONAL MEDIA ,07ry-u ^ nwn 
PRODUCERS COUNCIL 13/5 HAY -2 fill 1 1 : 27 



3150SPRING STREET 
FAIRFAX, VIRGINIA 22030 
(703) 273-7200 



EMTC BOARD OF DIRECTORS 

Chtirmtn: EDWARD J. M EEL L 
McGraw-Hill Films 
New York, Niw York 

Wc# Ctmlrrmn: S H ELTON STcR N 
DoubltdiV Multimedia 
Santa Ana, California 

IVAN BENDER 
'EBEC 
Chicago, tlllnoti 

SELMA BRODY 
MHItr-BfOdv Productions, Inc. 
Naw York, Naw York 

DAVID ENGLER 
Ganaral Educational Madia 
Graat Nack, Naw York 

Ex-Offhio: JOHN H. FISHER 
Guidanca AMOciatat 
Naw York, Naw York 

JOHN H. HUMPHREY 
Schloat Productions 
Tarrytown, Naw York 

SUZANNE T. ISAACS 
Sociaty for Vifual Education 
Chicago, Illinois 

HALKOPEL 
Ccrcnat Inttructional Madia 
Chicago, Illinois 

HERBERT LOBSENZ 
Markat Data Ratrlavat 
Wattport. Connacticut 

ROBERT F. O'REILLY 
Changing Timas Education Sarvica 
Washington, D. C. 

REGIONAL COORDINATORS 

EASTERN REGION 
JOHN H. HUMPHREY 
Schloat Productions 
Tarrytown. Naw York 

MIDWEST REGION 
HALKOPEL 
Coronat Instructional Madia 
Chicago, Illinois 

WESTERN REGION 
ARTHUR B. REYNOLDS 
Walt Disnay Educational Madia 
Glandala, California 



Mr, Kenneth Komoski 

Executive Director 

The Educational Products 

Information Exchange Institute 
463 West Street 
New York, New York 10014 

Dear Ken: 

At a meeting of the Board 'b± Directors of EMPC on April 14, 
1975, the Fifth Draft of EPIE's guidelines for "learner verifica- 
tion and revision" was reviewed and discussed. The representatives 
of EMPC who sat as unofficial observers/participants during the 
Strand I, Strand II, and Plenary Sessions expressed their views to 
the Board through Suzanne T. Isaacs, a Director of EMPC and Chair- 
woman of EMPC's Committee on Media Evaluation, 

The Board decided that option Three (3) in your March 21 memo 
best convey 's EMPC's position on these guidelines. That is: "we 
are totally dissatisfied with Draft #5 of the 'LVR Guidelines' and 
wish that our dissatisfaction, as expressed by the points outlined 
below, be appended to the guidelines when they .are distributed to 
all the interested parties," 

The shortness of time between the submission of Draft #5 and 
the need to render a decision does not permit a full explanation 
of our reasons. This fuller explanation will be forthcoming. In 
the meantime, EMPC wishes to make the following points: 



1. EMPC supports the use of appropriate evaluation 
techniques to determine the effectiveness of in- 
structional materials. However, there are a 
number of formative and summative evaluation 
methods currently available. EMPC feels it would 
be premature to identify, formalize and support 
something called "the LVR process" to the ex- 
clusion of all other evaluation methods. We feel 
this way for several reason's: 
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U There is a serious question about whether we can 
generalize about a positive relationship between 
formative evaluation and product improvement. 
Neither the "LVR process" nor any form of evaluation 
has demonstrated this* 

J. No single evaluation process can guarantee learning in 
all learning environments. 

EMPC would mislead educators and its own members by * 
even hinting that one system or process of evaluating 
instructional materials is superior to any other. 
Adoption of this position could lead to the very 
errors in selection of instructional materials we all 
are trying to avoid. 

. The interests of learners, educators, researchers and 
publishers are best served by using divergent modes 
of evaluation. 



E. Before adopting or supporting any process such as "LVR", 
it is necessary to show that it works; that it demon- 
strably improves instructional materials. Until this 

is done, we need to continue using the formative and sum- 
mative evaluation processes which are already in existence 
and which are continually being improved by educational 
researchers. 

F. The term, "learner verification and revision", really 
clouds the issue that what we are talking about is 
formative evaluation with some summative evaluation, 
EMPC feels it is necessary to work on improving forma- 
tive and summative evaluation rather than creating a 
new term. 

G. There is.no hint in Draft #5 that the ultimate decision 
to purchase and use instructional materials rests with 

the professional educator and not with the publisher telling 
the professional educator what to think, 

. Even as formative and summative evaluation methods are being 
employed and improved by Researchers and publishers, in- 
structional materials are a^lready subjected to possibly the 
most^^vigorous forms of testing. Namely, (a) "previewing" 
and "on-approval purchasing", and (b) actual classroom use. 
In the former case, educators may take any materials available 
into their classrooms and determine if they work with their 
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student population. The educators do not have to rely on 
the word of the publisher about how well the materials 
worked with some remote student group. In the lat-ter case, 
an even more stringent process is used. Once in the class- 
room, the materials one. used by prof es-sional educators with 
large numbers of. students in the actual environment for 
which the materials were intended. They are re-used, re- 
ordered or discarded on the basis of the results achieved. 
Since the environment changes from student to student, class 
• to class and school to school, this is the most empirical 
process possible and has provided the American schools 
with the widest variety and choice of instructional materials 
in the world. 

3. Unless and until the "LVR process" described in EPIE's 
Draft #5 is itself verified, EMPC is concerned that it 
can be misleading and damaging at the $tate and local 
level. The member companies of EMPC recognize that educa- 
tors require as much flexibility in evaluation of materials as 
producers need in creation; of these materials. The intro- 
duction of "learner verification laws" does nothing to change 
the fact that educators must always be responsible for eval- 
uation of materials which satisfy requirements of their own 
unique teaching situations. The EMPC calls for a re-evalua- 
tion of the existing legislation in California and Florida 
and urges any state and local system now contemplating 
"learner verification laws" or policies to seriously examine 
the experience in California and Florida bisfore adoption 

of such laws and policies. Laws should be the last in a 
long series of steps taken to iiiiprove evaluation knowledge 
and techniques. EMPC commits itself to assisting their 
colleagues in this process. 

4. FinaWy, the EMPC Board of Directors feels that the guide- 
lines^o not fairly or thoroughly reflect the serious doubts 
raised \y Task Force participants about "the LVR process." 

\ 

Even though EMPcVannot accept Draft #5, we believe our goals and 
yours are the same — better instructional materials for learners and 
educators. We both neeti^to continue to work on this matter with these 
goals in mind. Thank you\for asking for our reactions and permitting us 
to sit as unoff ici^ obserVers/participants on your Task Force. 



Sincerely, 

Daphne Philos 
Executive Director 



DP/n 
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ATTACHMENT #5 



To 

From 



Subject: 



Kenneth Komoski 



Susan S. Klein - - ^ - 

School Practice and Service Division 

Office of Dissemination and Resources 

(NIE Representative to the National Task Force on Learner Verification 
and Revision) 

Statement of Reservations on Draft #5—3/21/75 of the LVR Guidelines 

The comments that follow reflect my views and the views of a few 
other NIE staff members who took the opportunity to comment. (Draft 
#5 of the LVR Guideline was circulated to all NIE Task Force Leaders 
so that interested persons in the task force could contribute to 
this NIE response.) 

ye were generally satisfied with the introduction, orientation, and 
appendices but felt that the actual "Guidelines for Reporting and 
Assessing LVR Activities" need some major immediate revisions prior 
to more extensive exposure dnd that criteria for adequate or minimal 
adherence to the guidelines need to be suggested and explained. 
Thus, we checked response choice "b" and are expressing our praise 
and reservations. 



PRAISE 

We think that EPIE has been quite responsive in incorporating the 
ideas of the diverse constituent groups which it invited to partici- 
pate in the National LVR Task Force. Draft #5 also made progress 
in putting the LVR activity in perspective regarding major purposes 
(continued product improvement instead of product quality assessment) 
and complementary product improvement activities (rational instruc- 
tional design, care to promote social fairness, etc.). As R. Chesley 
from the NIE/ODR staff noted, "The process (LVR) described leaves a 
great deal to be desired and has a number of loopholes as mentioned 
(in following discussion). At the same time, it is probably better 
than not imposing any check on the material quality." 

RESERVATIONS 

Orientation Toward a Definition of the LVR Process 

1. Perhaps "publishers" should be defined broadly or used along 
with other terms such as "producers" or "developers." It is 
likely that many instructional materials developers or producers 
who do not consider themselves publishers would also find it 
beneficial to use LVR Guidelines. 
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2. Although progress has been made, additional emphasis can be 
placed on putting LVR in perspective relating to product 
purposes. (p, 1) it should be noted that (1) LVR does not 
guarantee that products will be effective with learners, but 
is apt to increase the potential for this effectiveness by 

• identifying some product deficiencies and (2) LVR does not 
generally address or claim to improve other dimensions of 
product quality such as cost effectiveness or social fairness 
(which may not be related to learner effects), 

3. We think that publishers should make assurances and claims 
about the quality and general predictive features and effects 
of their products when the products are used as specified. 
This consumer information is needed and desired. We agree 
that LVR activities and -information won't generally address 
itself to this issue. (p. 2) 

4. What might happen if the publisher states that the LVR process 
does not reasonably apply to the products, but the potential 
purchaser thinks otherwise? (p. 4) 

5. We agree that it is most likely that a producer will conduct 
LVR activities, but feel that evaluation data generated by 
users or external evaluator sources should also be used if 
available. These additional sources may be especially impor- 
tant in postpublication or other revisions performed by the 
producers. It is also possible that users could revise or 
adapt the materials if they receive the appropriate legal 
permissions. (You may have said some of this on p. 5 and 6 , 
but some of your statements appear contradictory.) 

6. If educators or others are to assess LVR reports common minimal 
criteria should be suggested. (p. 6) 

Guidelines for" Reporting and Assessing LVR Activities 

It may be most feasible to consider several of your categories as 
"catalog" information. These categories included: "Descriptive 
Information on the Product," "Instructional Design, "and "Conditions 
of Use of the Product." This. catalog information would be of general 
interest to future product selectors whether or not they are inter- 
ested in more detailed information on LVR, product effectiveness, social 
fairness, practicality, cos ts , etc. Desired information on categories 
such as instructional design should be clarified. 

The other categories could spell out the details of the LVR process in 
a supplemental LVR report. These categories and the detail requested 
seem to address two question areas: 
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1. Did the producers use an adequate rational procedure to 
collect feedback from appropriate learners, etc. and then 
make adequate revisions? This seems like the most important 
LVR screening question and is addressed by "Intended Learner 
Outcomes to be Investigated," "Techniques for Gathering Feed- 
back," "Description of Learners Usf^u in LVR Process," "Analysis 
of Findings and ^^Specif ic Improvements Made," and "Background 
and Future of Product." 

2. The second question is more of a research question to collect 
information to identify commonly needed types of revisions. 
Producers should not be forced to supply this information in 
detail unless it is needed for "1" above or for definite 
research projects. There should also be some attempt to deal 
with multiple feedback and revision cycles. Details on 
"Analysis of Findings" and "Specif icjj|Rnprovements Made" are 
likely to be helpful in product improvement research. Would 
findings such as "side effects" which are not related to 
intended learner outcomes be investigated? The first sentence 
under the "Specific Improvements Made" category is misleading. 
Producers can not always identify exactly where specific 
changes should be made by analyzing the findings although it 
does help if fine grain "item analysis" type data is examined 
rather than class mean gain scores. 



cc: C. L. Hutchins 
Senta Raizen 
Robert Chesley 
Art Wise 
Peter Briggs 



ATTACmENT # 6 
May 10, 1975 



Mr. P. Kenneth Komoskl 

The Educational .Products Information Exchange Institute 

463 West Street, 

New York 10014, New York 

Dear Mr. Komoskl, 

In response to ybur request for an evaluation of the Draft ffD 
of the LVR guidelines, I will answer as an Individual. Although 
I attended the task force In i\ew York - Strand II - In January 
as a member of the National School Board Association, It Is their 
policy that Individual school board members speak as only one 
representative and not as representing the official position of 
the NSBA. 

I have made my report to the NSBA and have forwarded to them 
subsequent copies of the drafts as they have been refined. 

To choose between a, b, or c as outlined In your letter of March, 
21, 1975, I would select (b) - 

(b) I am not completely satisfied with Draft if 5 of the LVH 
guidelines and ask that the statement of my reservations 
which I am attaching, be appended to it when it is dis- 
tributed to all interested parties. 

To summarize my opinion or objections to the LVk process and 
guidelines, there are f>' 'V points I would like to make. 

1. The concept that instructional materials should be tested 
on students or learners themselves to see if the students find 
difficulty using the materials is a good one. Perhaps I was wrou^ 

in my earlier assumption the responsible publishers did try uut their 
materials on the user-learner before release to the market place. 
As with anything, the true test of how well the LVR process is 
really attempted and how legitimate the results are as outlined 
or stipulated on the raoorting format, depends ultimately upon the 
integrity of the publisher and the sample group of learners used. 
In my opinion this process can produce better instructional materials 
or it can produce a very expensive process for all with increased 
costs to the consumer and very little tangible Improvements that 
would not already have been forthcoming. 

2, There is a dual responsibility between the publisher and 
"those charged with the responsibility of assessing the LVR 
reports" (as described on page 9). Here the Integrity and the 
reliability of the publisher is important as well as the purchaser 
of the materials understanding fully what the LVR process is sup- 
posed to evaluate - what it can do and cannot do. 
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This leads to my main reservation of the LVR process. Unless 
Individuals who use the LVR process^ both publisher and purchas<er 
-know and understand w hat the LVR process Is supposed to do. Its 
goals, and Its limitations, the "purchasers" will tend to expect 
more out of ±he process than It Is designed to do^^ or purchasers 
will assume, that certain results will be forthcoming, A quick and 
s-uperflclal mental translation by some Individuals of the meaning 
of 'Vlearner verification and revision" can mean that If this 
process Is used and materials are sold with this supportir^g data^ 
then "students will Learn - It literally verifies that students 
V/111 learn . " 

I make the strong suggestion that Appendix - Section G rfot\be 
the last page" of the document, but that It be Incorpora ted y t the 
very beginning In a concise Introductory paragraph that defAnes 
LVR a^nd Includes the clarification and the three points on page l6. 

3. The concept that we must not give students or learners any 
materials that they find difficult to use^ or offensive or 
prejudiced In any way, can lead to mediocrity of educational 
materials although this risk can be minimized. One problem that 
can be anticipated as this process grows In acceptance Is that 
there will be an Increase In the diversity of pressure groups and 
special Interest groups asking that the materials be L/lt'd to 
reflect their particular area of concern. 

The statement on page 6^ "if a school ex^pects that a publisher 
has put materials through the LVR process, that very same school 
must be ready to make It possible for the publisher to "LVR" that 
material..*. This can work two ways. If a school district wants 
the LVR process. It Is unfair to ask other schools to "carry the 
burden" (if there Is one) of using their classrooms, students^ 
teachers, time, etc. In this testing process. Also If one school 
district wants the LVR process to see how effective the materials a 
the resulting data might be .more meaningful.! If tried on their 
partfettlar type ofL-^s^mdeht body, l.js*^ materials LVRM on a metro-^ 
poll tan urba^'^cla ssroom might have different results from a rural' 
mldwestern student population. 'Different sections of the country 
or different student body compositions might require more than 
one LVR process to satisfy that area. This goes back to the basic 
que^stlon of the size of the sample and the type of student /learners 
tested . 

Kach year the school district I represent purchases thousands of 
items from text books to lab equipment to countlnc sticks. Where 
is the line drawn between "instructional material and a supply 
item. Some distinctions are easy to make^ but I suggest that 
on other items there could be a real question - is this an in- 
structional material item or a supply item. 
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The students are the ultimate consumer of our entire eaucational 
process and the materials- and teaching staffs are provided to. 
bring about the greatest growth possible in skills and knowleage. 
These "learners" should have the best materials and equipment 
we can devise and design and place in their hands at costs that 
are as reasonable as we can make them. Therefore, if this 
process can enhance this goal, then it should beco.Tie one part . 
of the process of selecting and evaluating instructional materials 
used in our classrooms. But, speaking aa one school \board member 
upon whom the ultimate responsibility falls to find the funas to 
balance all these needs, the add-on costs of the LVR brocess must 
Indeed prove to be worth the dollars Invested by the publishers 
and ultimately paid for by school districts in the puiAcnase price 
of the materials. • , 

blncerely, 

Karllyn Langdon 

bchool Board Member 

haclne Unified School Llstrict # 1 

Racine, Wisconsin 

Home address: 1032 Uorth Osborne Blvd. 

Racine, Vvlsconsln 53''-*03 



ERIC 
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ATTACUffiNT #7 ' 
Association of American Publishtrs, Inc. 



AndrM H. Nailly, Jr. Chairman 
Towmand Hoopat Prasidant 
Austin 4. McCaffiay Vice Prasidant 



One Park Avenue 
New York, N.Y. 10016 
Telephone 212^9-6920 
Cable»BOOK ASSOC NEWYORK 



May 1, 1975 



Mr. P. Kenneth Komoski 

The Educational Products Information 

Exchange Institute 
463 West Street 
New York, New York 10014 

Dear Ken: 

Thank you for sending the fifth draft of the Guidelines developed by the 
Task Force. The Research Committee of the'AAP continues to seek oppor- 
tunities to work within the publishing industryand the education 
profession to develop greater understanding and steps which can be taken 
to improve the quality of instructional materials. 

In recent months, we have had the opportunity to meet in review sessions 
with appropriate representatives, of the California Textbook Commission, 
California State Board of Education, members and representatives of the 
California State Department of Education, Members of the Committee have 
participated in both trade association workshops and in the annual meeting 
of the Association for Supervision and Curriculum Development. This 
continuing activity is consistent with the wish of AAP to "continue to 
be responsive to various requests from adoption units and independent 
agencies who wish to meet with representatives of the industry," as 
indicated in Dr. McCaffrey's letter to you on September 24, 1974. 
Consistent with that letter and the original position of the AAP School 
Division Executive Committee, we must respond to your letter of March 21, 
1975 by declining to indicate any one of your three options outlined. 
Rather as we stated in our September 24 letter, we feel we should "maintain 
AAP's role as an independent industry-wide association free to work with 
national, state and local agencies on validation and learner verification 
as each occasion requires." There are certain reservations that each 
member has as to both the content and the concept of LVR as expressed 
by draft five. I am sure that should they feel that clarification of 
their individual feelings are important in improving education, they will 
individually communicate these feelings to you. 

more 
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Finally, as our industry position paper stated, we contintie td feel that 
learner verification and revisicin must be considered only within the 
context of total materials development and selection procea^s. That position 
paper starts by recalling that ijjnerican publishers have a^ays assumed 
responsibility for developing effective learning materials and concludes 
with the important observation that, "The ultimate guarantee that publishers 
will continue to produce higher i quality materials is JLhe fact that educa- 
tional publishing is highly comjietitlve. Excellency is required in 
products-development in order to produce materials jdtiat will be accepted 
by professionally trained educators and used successfully by students." 

/ 



r 



Sincerely, 




. J^mes R. Squire 
^^hairman 
School Division 
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HOLT, mNEHART 
AND WINSTON 



CBS Inc., 383 Madison Avenue 
NewVbrt<, New Yofk 10017 
(212)6te-9l00 



June 6, 1975 



Mr. Kenneth Komoski 

Executive Directoy 

The Educational Products 

Information Exchange Institute 
M-63 West Street , . * 

New York, New York 10014 

Dear Ken: 

In response to your invitation to comment on your LVR 
ITiideline statement, I elect to choose option (c) : T 
am totally dissatisfied v/ith draft #5 of the LVR guide- 
lines and wish that my dissatisfaction, as I have 
expressed it in the g^ttached statement, be appended to 
it when it is distributed to all interested parties- 

During the past two yeirs. Holt has commited itself to 
an investment of time afid resources to develop , evaluate , 
and implement a workable Nnp^el for the collection of data 
from the people using its pt^ograms, both learners and 
teachers. In the 1974-75 year, our data collection 
efforts have brought us into contract with over 100 schools 
throughout the country. Our concern with the EPIE 
guidelines is that of a committed participant in the learner- 
based research and evaluation process. 

Somehow or another what promised to be a contribution to 
the improvement of education has become so confused that 
the impact of the "guidelines" will be more negative than 
positive . . . and that's a shame. Also it is a tragedy 
because it didn't need to turn out that way. Let's start 
with • some f uncjamental elements : 
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1. Improvement of education is the overall goal. 

2. The quality of instructional materials, as 
well as many other variables, i.e., teachers, 
facilities, support of parents contribute to 
achieving the overall goal. 

3 . Producers/developers/publishers of materials 
have a responsibility to produce better 
materials to help achieve the goal. 

The issue then comes to, "How do" publishers achieve this 
objective?" The "guidelines'^ seem to offer an answer, not 
to that question, but rather , "How . should publishers 
responsibly develop better materials?" Oh, I recognize 
that the statement warns the reader several times that 
"LVR" is not a panacea but that's in one breath, while 
everything else says, "This is the way>" And to make 
certain that no one misses the point the guidelines 
clearly suggest that if a publisher fails to follow the 
way , then those materials are unworthy of purchase. 
(Oddly, EPIE warns the reader to watch out for some publishers 
who do provide data in the suggested format for they may be 
"unscrupulous" or using "trickery''.) 

The commitment to a somewhat disguised, " the way " ; the 
continued commitment to an incomprehensible label "LVR" , 
seeming to prove once again that created terms that don't 
improve communication deserve to be discarded; the exploitation 
of human behavior to promote undue suspicion toward profit 
making companies (some of our stockholders would be amused 
with such a label) ; the total absence of responsible indication 
to the reader that if "the way" is implemented it will result 
in significantly higher costs of materials and in a reduction 
of the number of programs published (thus making development 
of materials possible for the major publishers and in turn 
eliminating competition) - all of this adds up to perhaps, 
a well-intentioned document that builds expectations which 
cannot be readily realized. 
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The question which comes to mind is, who has implemented, 
evaluated, and "verified" EPIE^s LVR process? We know 
that our own model requires ongoing assessment and adjust- 
ment; how wo;r?kable is the EPIE model? I suspect that 
EPIEi like the Publishing Industry , can make no guarantees. 
The state of the art of evaluation and testing limits us 
all. The EPIE guidelines assume the availability of 
instruments for collecting data from learners. Some data 
can be collected informally, but as yet, outside of 
achievement or criterion referenced testing, not very 
precisely or professionally. The simplest method, 
observation, is prohibitive in cost unless it is limited 
to a few sets of students within a limited geographical 
area. While attitudinal measures are being developed, 
their scope is often limited and many are still experimental. 

Now, let me say that concern about development and improvement 
of instructional materials is a valid one. Materials, like 
almost every other thing in life, can and should be better • 
Publishers have the central ^responsibility for building better 
materials. Such a responsibility includes the use of 
research at many levels - curriculum content, organization, 
manageability, and motivation. Inherent in the publishing 
process is the marshalling of resources to assure successful 
use. As EPIE rightfully indicates, children must be involved 
in this process for they are in most cases the final users. 
I say in "most cases" for there are many elements of an 
instructional material program which are directed towards 
the teacher who in turn is in fact the "delivery system". 
Thus the success of a total instructional sysLem is tied to 
materials/child; teacher/child, and/or teacher/materials/ 
child relationships. Research designs weighted exclusively 
to single relationships fail to provide adequate information 
to construct, or revise materials. 

Recognizing the state of the art of research and measurement 
generally and research on instructional materials specifically 
it seems to me that review committees would be better advised 
to begin with more reasonable questions than the guidelines 
propose. I say "reasonable" for I don^t think anyone can 
reasonably respond to the present format without generating 
disappointment and frustration. 
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Let me suggest two core questions: 

1. Publisher, how did you involve children and 
teachers in the development of this product? 
Explain . . . 

2. Publisher, now that the product is published, 
what is your commitment to involve children 
and teachers in aiding you to improve the 
product? Explain . . . 

School people have every right to expect better answers 
to these questions each year. A satisfactory answer 
today, will be less satisfactory a year from now, and 
still less and less two or more years hence. But such 
questions would represent a solid beginning to the 
original concern of EPIE, and all of us who believe 
that instructional materials are important but can be 
better. Yet at the same time we must maturely recognize 
the complexity of the problem. When President Lincoln, 
as reported in the EPIE statement , told his tailor to make 
his trousers long enough to reach the ground, they at 
least could agree on the purpose of trousers and where 
the ground was. Sadly, today, often times neither the 
school buyer, nor the publisher are at the equivalent 
level of understanding. Perhaps even less so, if they 
try to use the EPIE guidelines. 



Thank you for the opportunity to participate in some of 
the discussion on the issue. I assure you that Holt and 
most, if not all, other publishers are dedicated to the 
continued improvement of education through the development 
of increasingly better instructional materials. I believe 
EPIE shares that hope and promise. Yet somehow, you have 
over engineered the good idea. 





Thomas J. 
Vice President end General Manager 
School Department 
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ATIACHMENT #9_ 



Education Commission of tlie States 



,300 PNCOLN TOWER • 1860 LINCOLN STREET 
DENVER, COLORADO 80203 • (303) 893-5200 



April 4, 1975 

Personal statement by Chris Pipho to be attached to the Learner Verification 
and Revision guidelines. 

The notion that learners and teachers should have some involvement in 
the preparation, revision and updating of instructional materials is a good 
idea. I personally feel that reputable publishers and writers would not 
quarrel with this concept and in many ways are already carrying out the 
process. 

My personal opinion [because the Education Commission of the States 
has not discussed or passed a resolution on LVR] is that the idea, while 
noble, does not easily lend itself to legislative mandate. In order to 
properly carry out the concept, legislation would have to require that 
groups of people not usually bound together in the same governance- 
structured matrix, work together and be supervised by some state regulatory 
agency, i.e.: asking that publishers involve students and teachers without 
making 'provisions for local and state boards to preempt teacher and student 
time would be asking students and teachers to support the publishing indus- 
try. On the other hand, making publishers pay for the student and teacher 
time in an era of tight collective bargaining contracts could drive some 
publishers out of business. 

While I realize none of these fears can be substantially drawn from 
this document, the implementation and supervision of the process would 
eventually, in my opinion, call for increasingly more definitive legisla- 
tion. The end results I think would be of questionable cost-effective 
value. 

Up until now, a majority of the states have expressed no interest 
in learner verification and revision. The inference implied in the 
introductory section of the guidelines is that learner verification and 
revision is a coming trend. I do not agree with this contention. 



Respectfully submitted, 



Chris Pipho 

Associate Director 

Research and Information Services 

Education Commission of the States 

Denver, Colorado 
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AmCIMENT #10 



Association for Supervision 
and Curriculum Development 

kpAv 7, 1975 



Dr. Kenneth Baranski 
The Educational Products Information 

Exchange Institute 
463 West Street 
New York, NY 10014 

Dear Ken: 

• \ 

Thank you for the patience and extension of time. As per our April 7th 
telephone conversation. I have listed my cautions below: 

(1) that readers of the LVR guidelines note the temporary 
nature of the document as expressed in the task force desire 
to observe pilot situations; 

(2) that the pilot situations become a basis for possible 
review and update; 

(3) that readers are informed of the larger issues of 
instructional materials selection of which the LVR 
system is a part; 

(4) that readers are informed that predictive validity 
is not the intent of LVR; 

(5) that task force persons be reminded of the spirit 
of cooperation and openness rather than an unsung dis- 
trust; and, 

(6) that thi whole project continue to move slowly, 
deliberately, cautiously. 

Again, my thariks go out and my hat is off to you and Ken Komoski for 
your efforts to incorporate some of the above cautions into the introductory 
letter to be attached to the document. 

The document with a few editorial comments is attached for your consider- 
ation. Also, I have made the appropriate checks. It must be added that my 
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reserved approval reflects a personal view - not the view of the ASCD 
nance system* 

Again, thanks - good luck. Call on me as you need. 



Yours sincerely. 




Charles A. Speiker 
Associate Director 



CAS/mkt 
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Mr. P. Kenneth Komoski 
The Educational Producta 

Information Exchange Institute 
463 West Street 
New York, New York 10014 

Dear Ken: 

I apologize for failing to meet your deadline. My feelings 
on the fifth draft of your LVR report are. similar to my 
previous comments on the subject and I see no point in re- 
running those thoughts . 

With specific regard to the report itself, I believe it to 
be a lucid, well written doctament which more than adequately 
expresses your objectives for th6 program, but since I don't 
share your premise I can't bring myself to endorse the final 
work . 

I would be pleased to contribute anything I have to offer as 
you continue your efforts to refine this draft. I know I 
appear recalcitrant and accept the fact that I am. I^hope, 
nevertheless, that our difference never becomes one of more 
significance than our difference on this one issue. 



Sincerely, 




School Board Member, District #3 



EAT : ry 



